The main source code of cpprb is cpprb/PyReplayBuffer.pyx. Almost all the public classes and functions are implemented in this file. This file contains more than 2K lines (including blank lines), so that maybe we should divide it. However, if we want to utilize full cdef functionality, (as far as we know) we have to keep def in .pyx and cdef in .pxd separately. We feel that makes much harder to maintain and just put everything in the single file.
cpprb/VectorWrapper.pyx and cpprb/VectorWrapper.pxd have interface classes between .pyx and C++. These classes utilize buffer protocol, which enables for us to share large data without copy.
cpprb/ReplayBuffer.hh contains C++ implementation of PER sampling. In this file, there are a lot of template and SFINAE meta-programming. By using if constexpr, multi-thread (multi-process) version switches algorithm without runtime cost. In order to support multi-process shared memory, the classes can accept (shared) memory acquired at Python side.
cpprb/SegmentTree.hh is the one of the oldest code and has segment tree implementation. It can accept shared memory for multi-processing, too. The multi-processing version updates its tree lazily. Internal data structure is described at here.
cpprb/ReplayBuffer.pxd defines interface classes and functions of cpprb/ReplayBuffer.hh in order to use them in cpprb/PyReplayBuffer.pyx.
test/*.py have unit tests. A lot of regression tests are included, too. Dockerfile is used for build and test environment. .coveragerc defines coverage configuration for Coverage.py.
Under site directory, there are project site source and config file. sphinx directory contains main page and config file of API documentation. These documentation system is described “Documentation” section below.
benchmark and benchmark2 directory contain benchmark script and Dockerfile using at Benchmark page. These benchmarks might be outdated and need to be updated.
example directory has example codes. We need more examples.
CHANGELOG.org contains change log.
CITATION.cff is used for citation management at GitHub. Please read GitHub official blog post.
setup.py is used for package build. MANIFEST.in specifies package additional files.
.gitlab-ci.yml defines CI on GitLab CI/CD platform. .github directory contains CI config files for GitHub Actions. CI settings are described at “CI Configuration” section.
crean_reinstall.sh is old build script for local development. Recently, we don’t use it and we might remove it in the future.
Build system is defined at setup.py. We don’t use pyproject.toml nor requirements.txt. We know declarative programming is preferable, however, it cannot handle complicated conditional build rules, yet. For example, we would like to use Cython only when .pyx files exist and they are newer than corresponding .cpp ones. Source codes hosted at PyPI (not GitLab/GitHub) don’t include .pyx, then users who use pip command don’t need to prepare Cython at all.
In order to realize single command installation, NumPy and Cython are lazily imported after setup_requires become available.
Generated markdown of README is re-used as long_description describing PyPI package page. A unrosolved problem is that we cannot show the cpprb logo on the PyPI page.
Additionally, by setting DEBUG_CPPRB environment value, we enable debug mode build, where Cython line trace is enabled.
Project site are built by Hugo and hosted at GitLab Pages. The documents are written in Org Mode (site/site.org) and exported with ox-hugo. (aka. .org -> .md -> .html) This sequence is very useful to reuse (partial) README in the project site. The disadvantage is Org Mode is minor to markdown. site/init.el defines ox-hugo configuration.
We use Hugo Learn Theme. We also make custom shortcodes at site/layouts/shortcodes in order to create the highly customized landing page. We still need design improvement to make it easier and more understandable for new commers.
To show star and fork counts on the side bar, we use custom JavaScript defined at site/static/js/custom.js. We utilize GitLab REST API and GitHub REST API.
API reference is built with Sphinx from docstring at the source codes. The docstring adopts NumPy style, so that Napoleon extension is enabled. By using custom CSS (aka. sphinx/static/custom.css), we try to match the look and feel with that of the Hugo part. Additionally, we use sphinx-automodapi to build the module structure.
Unit test coverage report are hosted, too. The report is generated by unit test.
We mainly utilize GitLab CI/CD. We execute binary build, unit test, test coverage aggregation, PyPI upload, and project site build and deploy. The configuration is defined at .gitlab-ci.yml.
Unfortunately, schedule trigger jobs have not been working recently. We have to investigate it.
Additionally, GitHub Actions are used for specific purpose. Since we can use Windows and macOS for free, we execute test script on them and upload Windows binary to PyPI.
We utilize CodeQL for code security and quality analysis. The results are shown here (Is the page public?). To be honest, we are not sure whether this analysis is meaningful for us.